Towards Understanding Asynchronous Advantage Actor-critic: Convergence and Linear Speedup

Asynchronous and parallel implementation of standard reinforcement learning (RL) algorithms is a key enabler the tremendous success modern RL. Among many asynchronous RL algorithms, arguably most popular effective one advantage actor-critic (A3C) algorithm. Although A3C becoming workhorse RL, its theoretical properties are still not well-understood, including non-asymptotic analysis performance gain parallelism (a.k.a. linear speedup). This paper revisits algorithm establishes convergence guarantees. Under both i.i.d. Markovian sampling, we establish local guarantee for in general policy approximation case global softmax parameterization. obtains sample complexity $\mathcal {O}(\epsilon ^{-2.5}/N)$ per worker to achieve notation="LaTeX">$\epsilon$ accuracy, where notation="LaTeX">$N$ number workers. Compared best-known ^{-2.5})$ two-timescale AC, achieves linear speedup , which justifies asynchrony AC theoretically first time. Numerical tests on synthetic environment, OpenAI Gym environments Atari games have been provided verify our analysis.